The predictability of letters in written english

We show that the predictability of letters in written English texts depends strongly on their position in the word. The first letters are usually the least easy to predict. This agrees with the intuitive notion that words are well defined subunits in written languages, with much weaker correlations across these units than within them. It implies that the average entropy of a letter deep inside a word is roughly 4-5 times smaller than the entropy of the first letter. PACS: 89.70+c, 02.50.Fz, 05.45.Tp Since language is used to transmit information, one of its most quantitative characteristics is the entropy, i.e., the average amount of information (usually measured in bits) per character. Entropy as a measure of information was introduced by Shannon [1]. He also performed extensive experiments [2] using the ability of humans to predict continuations of printed text. This and similar experiments [3][4] led to estimates of typically ≈ 1 − 1.5 bits per character. In contrast, the best computer algorithms whose prediction is based on sophisticated statistical methods reach entropies of ≈ 2 − 2.4 bits [5]. Even this is better than what commercial text compression packages achieve: starting from texts where each character is represented by one byte, they typically achieve compression ratios ≈ 2, corresponding to ≈ 4 bits/character. These differences result from different abilities to take into account long-range correlations which are present in all texts and whose utilization requires not only a good understanding of language but also substantial computational resources. Formally, Shannon entropy h of a letter sequence (..., s−1, s0, s1, ...) over an alphabet of d letters is given by h = − lim n→∞ ∑ s −n,...,s0 p(s−n, ..., s0) (1) × log p(s0|s−1, ..., s−n) = lim n→∞ 〈− log p(s0|s−1, ..., s−n)〉 (2) where p(s−n, ..., s0) is the probability for the letters at position −n to 0 to be s−n to s0, and p(s0|s−1, ..., s−n) = p(s −n,...,s0) p(s −n,...,s−1) . The second line of this equation tells us that h can be considered as an average over the information of bit number. While Eq. (1) obviously assumed stationarity, we can define the latter also for nonstationary sequences, provided they are distributed according to some probability p which satisfies the Kolmogorov consistency conditions. The information of the kth letter when it follows the string ..., sk−2, sk−1 is thus defined as: